Draft: Initial draft implementation of CFG for LLM #996
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Based on the paper
Sanchez, Guillaume, et al.
"Stay on topic with Classifier-Free Guidance."
arXiv preprint arXiv:2306.17806 (2023).
a draft implementation of classifier free guidance.
This is simply for sharing internally and might very well be
completely wrong. It is debatable if we should expose such
a feature as a flag to the network or make it a separate
classifier instance (or a mixin). In the past we were
very much against special (potentially short-lived) feature
flags and it was much nicer to have this implemented as
an addon/callback. We might need to do something similar
here as well.
|
The paper in question is this one: https://arxiv.org/abs/2306.17806 Note that this method should have a greater effect the longer the labels are. Some random comments:
If this method works really well, I can see it being added explicitly. Alternatively, we could have a |
- Makes it possible to set gamma parameter - Setting it to `None` disabled functionality completely
ottonemo
force-pushed
the
feature/llm-classifier-free-guidance
branch
from
96de091
to
1c34aca
Compare
- `label_id` was misleading since it is actually a list of token ids related to a label and not a scalar value. Also the general process of generating logits it not related to labels at all but rather just to tokens - `kwargs` was named to be similar to transformers `generate` convention but is meant to be passed to `generate` and is therefore, in the context of `generate_logits` a model input. This should help the reader distinguish between expected input (`token_ids`) and model input (`model_input`)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Based on the paper
a draft implementation of classifier free guidance.
This is simply for sharing internally and might very well be completely wrong. It is debatable if we should expose such a feature as a flag to the network or make it a separate classifier instance (or a mixin). In the past we were very much against special (potentially short-lived) feature flags and it was much nicer to have this implemented as an addon/callback. We might need to do something similar here as well.
Open tasks: